Skip to content

Upgrade node delete timeout #5706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 18, 2025
Merged

Upgrade node delete timeout #5706

merged 1 commit into from
Jun 18, 2025

Conversation

@nojnhuh
Copy link
Contributor

@nojnhuh nojnhuh commented Jun 17, 2025

What type of PR is this?
/kind flake

What this PR does / why we need it:

This PR adds a nodeDeletionTimeout to control planes nodes in workload-upgrade tests to increase the tolerance for failed node deletions.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5705

Special notes for your reviewer:

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests
  • cherry-pick candidate

Release note:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 17, 2025
@k8s-ci-robot k8s-ci-robot added the kind/flake Categorizes issue or PR as related to a flaky test. label Jun 17, 2025
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 17, 2025
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 17, 2025
@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jun 17, 2025

This is [WIP] while it includes a hack to run more tests and changes from #5704.

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 17, 2025
@codecov
Copy link

codecov bot commented Jun 17, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 52.84%. Comparing base (8aa9eca) to head (549420a).
Report is 9 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #5706   +/-   ##
=======================================
  Coverage   52.84%   52.84%           
=======================================
  Files         278      278           
  Lines       29610    29610           
=======================================
  Hits        15647    15647           
  Misses      13146    13146           
  Partials      817      817

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

nojnhuh
nojnhuh commented Jun 17, 2025
View reviewed changes
templates/test/ci/patches/kcp-node-deletion-timeout.yaml
spec:
machineTemplate:
nodeDeletionTimeout: 60s
nodeDeletionTimeout: 600s
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking this timeout was starting when CAPI first tries to delete the Node, but it actually starts as soon as the Machine is deleted. So this has to account for cordoning and draining the Node and deleting the infra.

@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jun 17, 2025

/test pull-cluster-api-provider-azure-e2e-workload-upgrade

@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jun 17, 2025

Linux machine pool bootstrap flake. No logs so I can't really tell exactly what happened :/ Got past upgrading the control plane though which is what this is attempting to fix.

/test pull-cluster-api-provider-azure-e2e-workload-upgrade

@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jun 17, 2025

One more run and I'm convinced.

/test pull-cluster-api-provider-azure-e2e-workload-upgrade

@nojnhuh nojnhuh force-pushed the upgrade-node-delete-timeout branch from b4ed31b to 549420a Compare June 17, 2025 19:56
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 17, 2025
@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jun 17, 2025

This is cleaned up and ready for review.

/retitle Upgrade node delete timeout

@k8s-ci-robot k8s-ci-robot changed the title [WIP] Upgrade node delete timeout Upgrade node delete timeout Jun 17, 2025
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 17, 2025
mboersma
mboersma approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@mboersma mboersma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 17, 2025
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 02c8d7dcf96b1c1ef2bd794598b3e518cccb37c1

willie-yao
willie-yao approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@willie-yao willie-yao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: willie-yao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2025
@willie-yao
Copy link
Contributor

/retest

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jun 17, 2025 •
edited
Loading

@nojnhuh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cluster-api-provider-azure-capi-e2e b4ed31b link false /test pull-cluster-api-provider-azure-capi-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jun 17, 2025

/retest

@k8s-ci-robot k8s-ci-robot merged commit 02e5fba into kubernetes-sigs:main Jun 18, 2025
31 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Jun 18, 2025
@github-project-automation github-project-automation bot moved this from Todo to Done in CAPZ Planning Jun 18, 2025
@nojnhuh nojnhuh deleted the upgrade-node-delete-timeout branch June 18, 2025 02:14
@nojnhuh
Copy link
Contributor Author

nojnhuh commented Jul 15, 2025

/cherry-pick release-1.20

@k8s-infra-cherrypick-robot
Copy link

@nojnhuh: #5706 failed to apply on top of branch "release-1.20":

Applying: Add nodeDeletionTimeout to KCP machines for upgrade tests
Using index info to reconstruct a base tree...
M	test/e2e/data/infrastructure-azure/v1beta1/cluster-template-upgrades.yaml
M	test/e2e/data/infrastructure-azure/v1beta1/cluster-template-upgrades/kustomization.yaml
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/data/infrastructure-azure/v1beta1/cluster-template-upgrades/kustomization.yaml
CONFLICT (content): Merge conflict in test/e2e/data/infrastructure-azure/v1beta1/cluster-template-upgrades/kustomization.yaml
Auto-merging test/e2e/data/infrastructure-azure/v1beta1/cluster-template-upgrades.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 Add nodeDeletionTimeout to KCP machines for upgrade tests

In response to this:

/cherry-pick release-1.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@nojnhuh nojnhuh mentioned this pull request Jul 15, 2025
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mboersma mboersma mboersma approved these changes

@willie-yao willie-yao willie-yao approved these changes

@jackfrancis jackfrancis Awaiting requested review from jackfrancis

@marosset marosset Awaiting requested review from marosset

Assignees

@mboersma mboersma

@willie-yao willie-yao

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

CAPZ Planning
Status: Done

Milestone

v1.21

Development

Successfully merging this pull request may close these issues.

Occasional node deletion timeout causes workload-upgrade tests to fail

5 participants

@nojnhuh @k8s-ci-robot @willie-yao @k8s-infra-cherrypick-robot @mboersma